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Executive Summary 

Responding to calls for increased accountability from 
policymakers, accreditation agencies, and the public, 
colleges and universities are beginning to use evidence 
on what works to improve student success to design, 
manage, and improve educational programs and 
services. Building a “culture of evidence” to improve 
student success requires fundamental changes In the 
way that faculty, administrators, and support services 
staff use student data in decision making. 

Few empirical studies have been carried out on 
evidence-based decision making in the postsecondary 
sector. The study reported on here, conducted by 
researchers from the Community College Research 
Center and MDRC, examines what specific data college 
faculty and administrators use in their jobs and the extent 
to which they use data analysis to design and improve 
the impact of programs and services. (A full account of 
the study is given in the report, Evidence-Based Decision 
Making in Community Coiieges: Findings from a Survey 
of Faculty and Administrator Data Use at Achieving the 
Dream Coiieges, which is available for download at 
the CCRC website, http://ccrc.tc.columbia.edu.) The 
study is based on a survey on the use of student data 
by faculty and administrators at 41 community colleges 
participating In Achieving the Dream: Community 
Colleges Count. Achieving the Dream is a major national 
Initiative designed to improve educational outcomes 
for community college students, particularly students 
of color and low-income students. Twenty-seven of 
the surveyed colleges joined the Achieving the Dream 
Initiative in summer 2004 (Round 1), and 13 joined in 
summer 2006 (Round 3). In addition to the surveys, 
structured telephone interviews were conducted with 
administrators and faculty at four of the colleges that 
participated in the survey to explore In greater detail key 
findings from the survey. 

Achieving the Dream colleges use longitudinal student 
cohort data and other evidence to Identify gaps in 



student achievement. A key premise of Achieving the 
Dream is that once faculty and staff see that certain 
groups of students are not doing as well as others, 
they will be motivated to address barriers to student 
success. Achieving the Dream encourages participating 
colleges to broadly engage faculty, student services 
staff, and administrators in examining data on student 
progression, in prioritizing student achievement 
problems. In formulating strategies to address 
achievement gaps, and in evaluating the effectiveness 
of such strategies. Colleges are also expected to use 
evidence of what works to improve student success 
as the basis for academic program review, strategic 
planning, and budgeting, and thereby bring to scale 
and sustain proven strategies. 

The survey used in this study, which was administered 
by the Human Resources Research Organization 
(HumRRO), was conducted over a five-month period 
beginning in September 2007. It asked full-time faculty 
and administrators about what student data they use, 
how accessible data on students are at their college, 
how they use data in their jobs, and what types of data 
they find most useful. It also asked respondents about 
their familiarity and involvement with Achieving the 
Dream. The survey received a very favorable response 
rate, with 60% of faculty and 73% of administrators 
surveyed responding. In fall 2011 we plan to conduct 
another survey of the 1 3 Round 3 colleges In order to 
observe changes over time (the survey design allows 
us to match participants’ responses in 2007 with 
those in 2011). 

Results from the present study suggest that relatively high 
proportions of faculty and administrators at Achieving the 
Dream colleges use data on student outcomes: 

• More than half of surveyed faculty members 
reviewed or used data on placement test scores, 
retention rates, graduation rates, and measures 
of student learning other than grades at least 
once a year. 
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• About half of faculty and administrators used data 
on student achievement gaps on an annual basis. 

• About two thirds of faculty and administrators 
used outside research on effective practices at 
least annually. 

The high rate of data use at the colleges surveyed is 
perhaps not surprising, given their invoivement in an 
initiative premised on the importance of broadiy engaging 
faculty and staff in using data and research to improve 
the quality of programs and services: 

• Faculty and administrators who were Involved 
In Achieving the Dream used data on student 
outcomes more frequently and participated In 
organized discussions on Improving student 
outcomes much more frequently than did their 
colleagues who were not Involved In the Initiative. 

At the same time, not all participants in Achieving the 
Dream were heavy data users, and more generaiiy, 
there was considerable variation among individuai 
faculty and administrators in the extent to which 
they used student data. The teiephone interviews 
reveal that there may well exist a strong resistance to 
using data among some faculty. They also suggest 
that, despite survey findings that seem to indicate 
otherwise, many facuity and administrators may 
be uncomfortable in analyzing data. Contrary to 
expectation, we did not find much of a correlation 
between a facuity member’s rank or iength of time 
having worked at a coiiege and the extent to which he 
or she used data. 

In terms of an overarching commitment to evidence- 
based decision making, the findings suggest that there 
is only a limited connection between the extent of data 
use by faculty and administrators at Achieving the Dream 
colleges and the views and management practices of 
college leaders: 

• There Is only a weak correlation between various 
Indicators of data use and the extent to which 
respondents Indicated that their college overall 
uses data on outcomes to evaluate programs. 



• There Is little correlation between the extent 
to which administrators said that their college 
uses data for program-related decisions and the 
frequency with which they themselves used data. 

A central premise of Achieving the Dream is that 
commitment by a college’s leadership and the way that 
a college approaches program evaluation, strategic 
planning, and budgeting are key to encouraging the 
use of data for improvement by college personnel. 

Our findings suggest that leadership commitment 
and a data-oriented approach to institutional 
management may not in themselves be sufficient 
to encourage faculty and administrators to become 
more data-oriented in practice. Additional efforts at 
the department level are probably needed to change 
the behavior of faculty In particular. Indeed, we found 
that faculty In developmental education departments 
(and in for-credit occupational programs) were more 
frequent users of data than were faculty in other 
types of departments, particularly those in general 
education. This finding is not surprising given that 
improving developmental instruction has been a 
major focus of Achieving the Dream. It may well be 
that a similar intensive focus on improving outcomes 
is needed to change practices and to influence the 
culture in other departments. 

In terms of institutional-level results, contrary to our 
expectation, faculty and administrators at larger 
colleges were not, on average, heavier users of data 
than faculty and administrators at smaller colleges. 

And although colleges with higher levels of faculty 
and administrator participation in Achieving the Dream 
did not exhibit higher average rates of data use, 
we did find that colleges that joined Achieving the 
Dream in Round 1 had higher rates of data use than 
the Round 3 colleges on three of the four indicators. 

This is consistent with (although it does not prove) 
the hypothesis that colleges that have been involved 
in Achieving the Dream longer are more advanced in 
their use of data in improving student success. It also 
suggests that engaging faculty and staff in using data 
and building a culture of evidence is a complicated 
process that requires concerted effort over a long 
period of time. 
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1. Introduction and Background 

Over the past two decades, public education in the 
U.S. has been under increasing pressure to bring about 
and document improvements in student outcomes. 

The federal No Child Left Behind Act of 2001 requires 
states to monitor the performance of elementary, middle, 
and high schools based on standardized test scores 
and to remediate and, if necessary, reorganize or close 
schools that are unable to narrow student achievement 
gaps. Public higher education institutions have not been 
immune from the push for greater accountability. State 
and local policymakers, accreditation agencies, and 
boards of trustees are all seeking evidence of improved 
student outcomes. 

An increasing number of public schools, colleges, and 
universities are responding to these demands. Borrowing 
from management techniques first employed in private 
industry and capitalizing on advances in information 
technology, many are beginning to use data analysis 
to design, manage, and improve educational programs 
and services. Instituting a “data-driven” or “evidence- 
based” approach to decision making aimed at increasing 
student success requires fundamental change in the way 
that these educational institutions operate. Proponents 
sometimes refer to this comprehensive organizational 
change process as “building a culture of evidence.” 

There are only a handful of larger-scale empirical studies 
on evidence-based decision making in higher education. 
In one study, Petrides and Nodine (2005) examined how 
faculty, administrators, and staff in a community college 
district in California used student outcomes data for 
decision-making. In 2000, the researchers interviewed 
70 college personnel and administered a survey to 
220 faculty, administrators, and staff. In 2002, they 
conducted another study of data use in a large, urban 
community college district. This time they interviewed 27 
faculty, administrators, and staff about how data retrieval 
had changed since the implementation of a decision 
support system and a data warehouse in 1998. Their four 
key findings from these two studies are that 1) external 
accountability mandates do not necessarily result in 
improved data use; 2) even when there are “information 
bottlenecks” at a college, people find “workarounds” 
that include manual data collection and manipulation 



and local database creation; 3) even in districts that are 
committed to a data-driven environment, significant 
barriers remain, which include concerns about the ability 
to use technology and data, concerns about “information 
overload,” and concerns that data are perceived as 
unreliable; and 4) the “information culture” and the 
support for evidence-based decision making within 
districts is influenced by the district leadership. 

In another study, Goldstein (2005) presented results of 
a national survey of senior IT managers at more than 
380 postsecondary institutions on the use of data, the 
technology and tools that support data collection and 
analysis, and the degree to which data support decision 
making. The study found that the use of data in decision 
making about college programs and practices, or what 
are called “academic analytics,” provides colleges 
and universities with a competitive advantage, at least 
as perceived by the interview subjects. The research 
suggests that the factors most strongly associated 
with an institution’s perceived success in implementing 
academic analytics are training, leadership commitment, 
and staff with strong analytical skills. 

None of the empirical research to date has examined 
in depth what specific data college faculty and 
administrators use in their jobs and the extent to which 
they use data analysis to design and improve the impact 
of programs and services. The report presented here 
offers findings from a study designed to fill that gap in 
the knowledge base. The study was based on a survey 
and on telephone interviews about the use of student 
data by faculty and administrators at community colleges 
participating in Achieving the Dream: Community 
Colleges Count. Achieving the Dream is a major national 
initiative designed to improve educational outcomes for 
community college students, particularly students of 
color and low-income students. More information on the 
initiative is available at www.achievingthedream.org. 

Achieving the Dream encourages colleges to undertake 
the following five-step process to bring about changes in 
policy and practice that lead to improved student success: 

Step 1: Commit to improving student outcomes. The 
college’s senior leadership, with support from the board 
of trustees and faculty leaders, commits to making the 
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changes in policy and resource allocation necessary to 
improve student outcomes, communicates the vision 
widely within the college, and organizes teams to 
oversee the process. 

Step 2: Use data to identify and prioritize problems. 
The college uses longitudinal student cohort data and 
other evidence to identify gaps in student achievement. 
A key premise of Achieving the Dream is that once 
faculty and staff see that certain groups of students 
are not doing as well as others, they will be motivated 
to address barriers to student success. To ensure that 
they utilize their resources to greatest effect, colleges 
are encouraged to prioritize the student achievement 
problems that they plan to address. 

Step 3: Engage stakeholders in developing strategies 
for addressing priority problems. The college 
engages faculty, staff, and other internal and external 
stakeholders in developing strategies for remedying 
priority problems with student achievement, based 
on a diagnosis of the causes and an evaluation of the 
effectiveness of previous attempts by the institution 
and others to address similar problems. 

Step 4: Implement, evaluate, and improve strategies. The 
college then implements the strategies for addressing 
priority problems, being sure to evaluate the outcomes 
and using the results to make further improvements. 

Step 5: Institutionalize effective policies and practices. 
The college takes steps to institutionalize effective 
policies and practices. Attention is given to how 
resources are allocated to bring new initiatives to scale 
and to sustain proven strategies. The processes of 
program review, planning, and budgeting are driven by 
evidence of what works best for students. 

Achieving the Dream expects that by following these 
steps, colleges will be able to build a “culture of inquiry 
and evidence” that will lead to continuous improvements 
in student success. 

Achieving the Dream’s five-step model shares principles 
with the Total Quality Management (TQM) movement 
and other quality improvement programs. The quality 
improvement movement in postsecondary education 



is exemplified by models used in the Academic 
Quality Improvement Program of the Higher Learning 
Commission of the North Central Association of 
Colleges and Schools, the Quality Enhancement Plan 
of the Southern Association of Colleges and Schools, 
and by the Equity Scorecard initiative (Bensimon, 

2004, 2005; North Central Association of Colleges and 
Schools, Higher Learning Commission, 2008; Sallis, 
2002; Southern Association of Colleges and Schools, 
Commission on Colleges, 2008). Each of these models 
includes a process of data gathering to yield a more 
informed understanding of problem areas, a selection 
of solutions based on the data, and the adoption of an 
organizational feedback loop in which processes are 
continuously evaluated and improved. 

Qver 80 colleges in 15 states are currently involved 
in Achieving the Dream, which is supported by 
Lumina Foundation for Education and other funders. 
Participating colleges are asked to undergo a year- 
long process of data analysis and planning, followed 
by a four-year process in which they are expected 
to implement, evaluate, and, where appropriate, 
bring to scale strategies for improving student 
success. Achieving the Dream provides financial 
support to the colleges, including a planning grant 
and an implementation grant totaling $450,000 over 
five years to support data collection and analysis 
and implementation of improvement strategies. A 
coach and data facilitator are assigned to help each 
institution, and colleges receive further guidance 
through participation in an annual strategy institute, 
where they share experiences and findings with 
other colleges. 

In the study reported on here, researchers surveyed 
full-time faculty and administrators at 41 Achieving the 
Dream colleges in 7 states. These included 28 colleges 
that joined the initiative in the first round (in summer 
2004) and 13 colleges that joined in the third round (in 
summer 2006). The survey asked full-time faculty and 
administrators about what student data they use, how 
accessible data on students are at their college, how 
they use data in their jobs, and what types of data they 
find most useful. The survey also asked respondents 
about their familiarity and involvement with Achieving 
the Dream, and it asked for their views more generally 
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on reform efforts at their college to improve student 
success. The survey was conducted over five months 
beginning in September 2007— approximately three 
years after the Round 1 institutions began their work 
and one year after the Round 3 institutions began theirs. 

It received a very favorable response rate, with 60% of 
faculty and 73% of administrators surveyed responding. 
To our knowledge, this is the most extensive survey to 
date on the use of data and attitudes toward data use by 
college faculty and administrators. In fall 201 1 we plan 
to conduct another survey of the 13 Round 3 colleges in 
order to observe changes over time (the survey design 
allows us to match participants’ responses in 2007 with 
those in 2011). 

In addition to the surveys, structured telephone 
interviews were conducted with administrators and 
faculty at four of the colleges that participated in the 
survey. The purpose of these interviews was to explore in 
more depth key findings from the survey. 

This report is organized as follows. Section 2, following 
this introduction, presents the methodology used in this 
study. Section 3 describes the main findings on general 
patterns of data use by faculty and administrators at 
the Achieving the Dream colleges. Section 4 examines 
factors associated with the use of student outcomes data 
by faculty and administrators. Section 5 explores why 
data use by faculty and administrators is more extensive 
at some colleges than at others. Section 6 presents 
findings from the in-depth telephone interviews with 
faculty and administrators at four of the colleges. The 
final section, 7, presents our conclusions and discusses 
the implications of efforts by community colleges to 
improve student success. 

2. Methodology 

This section describes the methodology used for the 
survey of faculty and administrators and for the in-depth 
telephone interviews; these were the main sources of 
data for the study. 

Survey of Faculty and Administrators 

To examine patterns of and reasons for data use by 



faculty and administrators at Achieving the Dream 
colleges, we designed a survey to be delivered using 
self-administered, online instruments. 

The CCRC/MDRC research team developed two survey 
instruments through a process involving extensive internal 
review and revision. The instruments were pilot tested 
in spring 2007 with faculty and administrators at three 
second-round Achieving the Dream colleges, and were 
further revised based on the responses and feedback from 
institutional researchers at the pilot colleges. 

The administrator instrument contains 100 items and 
the faculty instrument contains 101 items, most of 
which overlap. Both instruments ask questions on 
three main topics: 1) data accessibility; 2) data use and 
usefulness, and 3) familiarity with and involvement in 
Achieving the Dream. Most items use a 7-point Likert 
scale, although some are yes/no questions. Both 
instruments also ask for demographic information about 
the respondent and include an open-ended question 
about both the use of data at the respondent’s college 
and views on Achieving the Dream. 

The sampling frame included administrators and full-time 
faculty members at the colleges that joined the Achieving 
the Dream initiative in Round 1 (in summer 2004) and 
in Round 3 (in summer 2006). Three of the Round 1 
colleges declined to participate in the survey, leaving 28 
Round 1 colleges in five states (Florida, New Mexico, 
North Carolina, Texas, and Virginia) and 13 Round 3 
colleges in two states (Pennsylvania and Washington). 

The survey was administered by the Human Resources 
Research Organization (HumRRO). HumRRO also 
participated in the final stages of questionnaire 
development. They were responsible for the web 
programming to permit online administration of the survey, 
maintenance of the online survey, management of the paper 
versions of the survey and of the telephone administration 
of the survey. They also served as the primary contact 
for survey questions and technical difficulties. In January 
2008 they provided CCRC with an interim dataset, and in 
February 2008 they provided the final dataset. 

We asked colleges to provide names, titles, and 
email addresses for full-time faculty members and 
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administrators. For the purposes of this survey, we 
defined “administrator” as anyone at the assistant/ 
associate director ievel and above, but we allowed the 
colleges to decide whom to include in this category. In 
cases where a college provided us with all non-faculty 
positions, we edited the list of administrators based on 
the job titles provided to us. This generated 2,209 full- 
time administrators. 

We confined the faculty survey to full-time faculty 
members, since many colleges do not collect contact 
Information on adjunct faculty at the college level. Due to 
the large number of faculty In some of the larger colleges 
Involved, we used a single-stage sampling procedure 
to reduce the number of faculty who were Invited to 
participate in the study. We randomiy selected 150 faculty 
members at the colleges with faculties larger than 1 50 
by automatically generating random numbers for each 
Individual and then selecting the first 150, the minimum 
number we estimated was needed to ensure statisticai 
rigor. We deieted individuais who were no ionger affiliated 
with a given coliege or were on a ieave and seiected the 
next person on the iist as a replacement. This process of 
replacement continued oniy during the pre-survey phase, 
when we were testing email addresses; once the survey 
was deployed, no replacements occurred. Through this 
process, we Invited 4,130 full-time faculty members to 
participate in the study. 

The survey was administered beginning in September 
2007. To encourage a high response rate, we used 
a multi-phase administration process, it began 
with a preliminary emaii designed to give advance 
notice of the survey and to test the list of emaii 
addresses we had received from the colleges. The 
next communication, one week later, was an email 
with the URL for the survey and a unique password 
for each respondent. The third email was sent as a 
follow up one week later to everyone who had not 
yet responded. A fourth email was sent one month 
after the initial invitation. One week later, letters with 
a hard copy of the survey enclosed were mailed to all 
non-respondents. The mailing of the hard copy letters 
coincided with a request to the college presidents to 
encourage their faculty and administrators to complete 
the survey. Phone administration of the survey to 
those who had not yet responded began just over two 



months after the initial email was sent. At this time, 
we were approaching winter break, so we suspended 
efforts to contact the faculty and administrators at 
the colleges until after the new year. We sent a final 
email to all non-respondents and finished the survey 
with a final one-week phone administration in the 
week following the February 2008 Achieving the 
Dream Strategy Institute, which teams from every 
participating college attended. 

The survey achieved a response rate of 60% from the 
faculty surveyed and a 73% response rate from the 
administrators. Response rates varied considerably by 
college. Responses to the faculty survey ranged from a 
low of 37% at one college to a high of 85% at another. 
College response rates to the administrator survey 
ranged from 57% to 93%. Response rates by college 
are provided in the full report of this study. 

Respondent demographics of faculty and 
administrators were fairly similar across colleges. Of 
the faculty respondents, 56% were female, and of 
the administrators, 62% were female. The majority 
of faculty and administrator survey respondents, 
more than 78%, identified themselves as White. The 
next largest group comprised those who identified 
themselves as Hispanic/Latino/Spanish; they made 
up 12% of the faculty and 15% of the administrators. 
Black, Asian or Pacific Islander, and American Indian 
or Alaskan Native faculty comprised 8%, 3%, and 
2% of the faculty respondents, respectively. Black 
administrators made up 12% of the administrator 
respondents. Asian or Pacific Islanders and American 
Indian or Alaskan Natives made up 3% and 1 % of 
administrator respondents, respectively. Ten percent of 
faculty and 8% of administrators identified themselves 
as “other.” 

More than 62% of the faculty who completed the 
survey taught In academic transfer/general education 
areas. Administrators Involved In academic affairs and 
instruction made up 20% of the respondents, and those 
in student affairs/student services accounted for 18% of 
the respondents. The institutional research/effectiveness 
staff accounted for just over 3% of the respondents. 
Additional demographic information Is provided in the full 
report of this study. 
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In-Depth Telephone Interviews 

To explore the survey findings on the patterns and 
determinants of data use in greater detaii, we conducted 
in-depth teiephone interviews with key personnei at four 
colleges that participated in the survey. 

These four coiieges were seiected on the basis of four 
criteria. First, the coiieges are all first-round participants in 
the initiative. This means that they have been exposed to 
the concept of a culture of evidence for almost four years. 
This was important because creating a culture of evidence 
requires fundamental organizational change, so colleges 
needed to have had time to adopt such a culture. 

The second and third criteria we used influenced the 
selection of the states in which the colleges are located. 
Note that the Round 1 Achieving the Dream colleges 
were drawn from five states: Florida, New Mexico, North 
Carolina, Texas, and Virginia. The states chosen for 
telephone interviews were selected on the basis of their 
accrediting region. New Mexico and Virginia were selected 
because they are in regions with accrediting agencies that 
were the earliest to adopt guidelines relating to the use 
of data and evidence to improve student outcomes. New 
Mexico is under the North Central Association of Colleges 
and Schools’ Higher Learning Commission (NCA-HLC). 
Virginia is a member of the Southern Association of 
Colleges and Schools (SACS). NCA-HLC and SACS were 
the first of the regional accreditation agencies to have 
focused attention on the use of evidence for improving 
student outcomes (Biswas, 2006). We hypothesized that 
the longer the accrediting region had these guidelines 
in place, the stronger the pressure on colleges in these 
states would be to use data to support improving student 
outcomes. Also, both states have adopted performance 
funding policies within the past three years. Such policies 
require colleges to report performance data, and they 
tie a certain percentage of funding for colleges to their 
performance. Together, these criteria were important 
because they suggest that these are states in which 
colleges are strongly encouraged to use data to document 
and improve performance. 

Finally, the colleges selected represent a range of 
sizes within each state. We hypothesized that smaller 
colleges would have an easier time promoting data use 



as part of the initiative because there would be fewer 
communication and coordination issues. 

Once the colleges were selected we identified potential 
interviewees at each college based on the functions they 
carried out. They did not need to be active participants in 
Achieving the Dream. Persons we interviewed included 
the administrative leadership of the college, including 
the president and vice presidents, the Achieving the 
Dream coordinator, institutional research staff, information 
technology staff, student services personnel, and a mix of 
faculty teaching developmental and non-developmental 
education courses. We created interview protocols for 
each position and asked questions about interviewees’ 
use of data, their perceptions about the decision-making 
process at the college, and the role of Achieving the 
Dream and other initiatives in supporting data-based 
decision making at each college. These interviews were 
conducted in the summer and fall of 2008. 

We interviewed 1 7 faculty members and 25 administrators. 
The interviewees included a mix of individuals who had 
and had not been directly involved with the Achieving the 
Dream initiative. The faculty we interviewed taught either 
developmental or college-level math or English. The senior 
administrators we interviewed included, at each college, 
the senior academic officer, the senior student services 
officer, the director of institutional research, the Achieving 
the Dream coordinator, and at least one member of the 
student services staff. 

3. Patterns of Data Use by 
Faculty and Administrators 

This section presents descriptive statistics of survey 
responses from faculty and administrators at Achieving 
the Dream colleges to questions concerning the use 
of data and research on students by themselves, their 
departments, and their colleges. 

Use of Data by Faculty and Administrators 

Frequency of data use by type. Use of student 
outcomes data by faculty and administrators at the 
surveyed Achieving the Dream colleges was perhaps 
more widespread than expected based on our earlier 
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findings in a baseline evaluation of first-round Achieving 
the Dream colleges (Brock et al., 2007) and based on 
feedback from the coaches and data facilitators about 
the colleges’ use of data and research. Table 1 shows the 
percentages of faculty and administrators who Indicated 
using or reviewing various sorts of information at least 
once a year as well as the percentage who said they 
never use a given type of information. Not surprisingly, 
high percentages of faculty reviewed or used grades 
and course evaluations at least once a year, while the 
majority of administrators reviewed information on 
college finances at least annually. What might be more 
unexpected is that more than half of faculty used data 
on placement test scores, retention rates, graduation 
rates, and measures of student learning other than 



grades, among other types of student outcomes data, 
once a year or more. Still, over a third of faculty never 
used measures of student learning other than grades and 
never looked at Information on students broken down by 
race or ethnicity, and nearly a third never reviewed data 
on student achievement gaps. 

Most faculty and administrators reviewed enrollment 
data at least once a year. About two thirds of faculty 
and administrators used outside research on effective 
practices at least annually. About half of faculty and 
administrators reviewed or used data on student 
achievement gaps on at least an annual basis. 
Administrators were more likely than faculty to use 
research produced by their own college. 



Table 1 

How often do you review or use the following types of information in your job? 




Percentage of Respondent Group Using the 
Given Information Type* 


At least once per year 


Never 


Type of Information 


Faculty 


Admins. 


Faculty 


Admins. 


Placement test scores** 


69.3 


49.4 


23.8 


44.9 


Enrollment data 


87.3 


83.6 


8.8 


14.0 


Grades 


89.9 


61.3 


7.1 


34.3 


Course evaluations 


89.4 


46.4 


5.3 


47.6 


Measures of student learning 


54.7 


45.5 


34.3 


47.0 


other than grades** 










Retention rates** 


72.0 


65.3 


17.5 


29.0 


Graduation rates** 


64.5 


64.0 


22.7 


29.5 


Transfer rates** 


46.4 


50.5 


35.6 


40.7 


Percentage of students successfully completing developmental education** 


47.8 


48.4 


37.9 


42.9 


Financial aid** 


35.2 


52.6 


52.1 


39.1 


College budget and finances 


51.2 


75.3 


34.9 


18.9 


Results from external surveys** 


41.8 


51.7 


39.2 


36.9 


Focus groups or other qualitative data** 


40.1 


51.5 


37.1 


33.2 


Research by the college** 


58.1 


68.2 


21.5 


20.2 


Outside research on effective practices 


69.0 


65.9 


16.4 


23.4 


Data on student achievement gaps** 


50.7 


49.7 


30.8 


39.1 


Information broken down by students’ race or ethnicity** 


47.0 


58.7 


35.6 


31.7 


Information broken down by students’ income levels or receipt of financial aid** 


32.2 


48.5 


51.7 


41.1 



* “Type of Information Not Available" responses were treated as missing. 
** Types of data that are promoted by Achieving the Dream. 
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Perceived usefulness of data by type. The majority 
of faculty and administrators surveyed found most of 
the types of information listed below in Table 2 at least 
somewhat useful In their jobs. A slightly higher share 
of administrators than faculty found these types of 
Information at least somewhat useful in their jobs, but 



the percentages are generally high for both faculty and 
administrators. Two thirds of the faculty respondents 
Indicated that research reports and other information 
their college provides are generally helpful to their work 
as teachers. 



Table 2 

In your role as a faoulty member (administrator), rate the usefulness of the following types of 
Information to your job. 




Percentage of Respondent Group Indicating Type of 
Info is “Somewhat Useful” to “Very Useful”* 


Type of Information 


Faculty 


Administrators 


Placement test scores 


74.1 


83.4 


Enrollment data 


75.1 


94.8 


Grades 


84.7 


85.4 


Course evaluations 


89.9 


82.9 


Measures of student learning other than grades 


70.9 


82.0 


Retention rates 


80.6 


89.5 


Graduation rates 


70.8 


87.7 


Transfer rates 


65.3 


83.1 


Percentage of students successfully completing developmental education 


63.7 


82.9 


Financial aid 


41.9 


81.8 


College budget and finances 


50.3 


92.9 


Results from external surveys 


54.0 


81.3 


Focus groups or other qualitative data 


60.4 


83.8 


Research by the college 


66.7 


88.6 


Outside research on effective practices 


79.1 


87.6 


Data on student achievement gaps 


69.3 


82.1 


Information broken down by students’ race or ethnicity 


50.9 


80.6 


Information broken down by students’ income levels or receipt of financial aid 


45.9 


80.0 



* “Not Applicable” responses treated as missing. 
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teaching. Around one in five is a heavy user of data and 
research for teaching decisions. Oniy around one in ten 
use data and research “not at ail.” 



Use of data and research by faculty in teaching-related 
decisions. As is evident from Table 3 below, the majority 
of faculty at Achieving the Dream colleges use data and 
research at least to some extent in decisions related to 



Table 3 

How much do you use data and research on students in your own decisions about the following? 




Percentage of Faculty Who Use Data and Research for 
the Given Decision Type 


Decision Type 


At least some 


A lot 


Not at all 


Curriculum 


76.2 


18.3 


14.1 


Teaching practices 


82.2 


24.2 


9.0 


Advising students 


80.7 


23.3 


11.3 


Identifying students who are struggling academicaiiy 


78.2 


22.3 


11.2 



Participation in organized discussions on improving 
student success. The majority of faculty and 
administrators at Achieving the Dream colleges indicated 
that they participate at least once a year in organized 
discussion on improving student success (see Table 
4). Fewer participate in discussions about the needs 
or performance of students of color or of low-income 
students in particular than about improving academic 
achievement of students more generally. Approximately 



one third of faculty respondents indicated that they never 
participate in discussions about the performance of 
students of color or low-income students. 

In a question not reflected in Table 4, two thirds of faculty 
respondents agreed that asking faculty members to 
regularly participate in discussions about data on student 
outcomes is a good use of their time. Fewer than 20 
percent (18%) disagreed with this. 



Table 4 

How often have you participated in organized discussions at the college on topics related to 
improving student success? 




Percentage of Respondent Group Participating in 
Discussions on the Given Topic 




At least once per year 


Never 


Topic of Discussion 


Faculty 


Admins. 


Faculty 


Admins. 


Improving academic achievement or closing achievement gaps 


78.0 


63.9 


9.5 


26.8 


Academic needs or performance of students of color 


56.2 


57.0 


26.5 


34.4 


Academic needs or performance of low-income students 


55.9 


59.2 


26.5 


32.4 
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Use of data by academic departments. Most faculty 
who responded to the survey indicated that they 
were in departments that use data and research for 
programmatic decisions at least to some extent (see 
Table 5). Around one in four were in departments that 
are heavy users of data to make program decisions. 
Only a small percentage of the faculty respondents 
were in departments that do not make use of data 



and research for such decisions. 

In a question not reflected in Table 5, four out of five faculty 
respondents indicated that their department meets at 
least one a year to discuss the success rates of students 
in the courses they teach. A similar majority (80.2%) are 
in departments that regularly evaluate the effectiveness of 
new academic programs, projects, or practices. 



Table 5 

How much does your department use data and research on students in decision-making about the 
issues below? (Question asked of faculty only.) 




Percentage of Faculty Whose Department Uses Data and 
Research for Decisions on the Given Topic* 


Issues 


At least some 


A lot 


Not at all 


Curriculum 


79.2 


25.2 


11.1 


Teaching practices 


81.5 


22.6 


9.6 


Tutoring or other academic support 


79.4 


22.3 


10.4 


Program pianning 


83.0 


25.5 


7.7 


Academic program review or evaluation 


84.5 


27.0 


6.9 


Long-term strategic planning 


80.4 


22.4 


8.8 


Budgeting or resource allocation 


71.1 


17.6 


14.6 


Identifying and redesigning high-failure-rate courses 


51.8 


21.5 


15.2 



* “Don’t Know” responses treated as missing. 



Use cf data by ccllege. The survey asked 
administrators to assess how much their college uses 
data and research on students In decision-making. 

As is evident from Table 6 (page 14), the majority of 
respondents Indicated that their college uses data and 
research on students in decision-making on program 
and planning issues at least to some extent. A third 
or more indicated that their college uses data and 
research extensively. Only a small fraction indicated 
that their college does not use data and research In 
decision-making. 



In a question not reflected in Table 6, the vast majority of 
administrators (91.0%) also Indicated that their college 
uses data on student outcomes (e.g., persistence, 
learning, or degree attainment), not just enrollments, to 
evaluate academic programs and departments. A similar 
percentage (92.5%) indicated that each department or 
division in their college is required to set measurable 
goals and objectives as part of the planning process. 
Three fourths (75.3%) of administrators said that budget 
requests at their college must be supported by evidence 
that students will benefit as a result. 
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Table 6 

How much does your college use data and research on students In decislon-making about the 
Issues below? (Question asked of administrators only.) 




Percentage of Administrators Indicating Their College 
Uses Data and Research for Decisions on the Given Topic’* 


Issue 


At least some 


A lot 


Not at all 


Curriculum 


88.1 


31.6 


4.5 


Program planning 


60.1 


32.9 


3.6 


Academic program review or evaluation 


56.6 


36.5 


3.2 


Long-term strategic planning 


55.0 


38.6 


3.2 


Budgeting and resource allocation 


55.5 


36.2 


4.2 


Identifying areas for improvement at the college 


55.4 


38.3 


2.7 



* “Don’t Know” responses treated as missing. 



Influence on the use of data. The survey asked 
administrators about the sources of influence on the use 
of data by themselves and their colleagues at the college. 
As shown in Table 7, the majority of administrators 
responding to the survey believed that accreditation, 
Achieving the Dream, and the college’s senior leadership 



have had a lot of influence on data use at their institution. 
A near majority believed that state mandates and other 
grant programs have had a lot of influence on data use at 
their college. Only a quarter of administrators indicated 
that their college’s trustees have had a lot of influence on 
the use of data. 



Table 7 

How much Influence have the following had In promoting data use by college administrators at 
your college? (Question asked of administrators only.) 




Percentage of Administrators Indicating That the Given 
Possibility Has Had Influence on Data Use at College’* 


Possible Influences 


At least some 


A lot 


Not at all 


College leadership (chancellor, presidents, vice presidents) 


92.9 


56.0 


2.7 


College board of trustees 


73.8 


25.4 


11.9 


state mandates 


93.5 


46.6 


2.5 


Accreditation reviews 


97.2 


63.6 


1.4 


Achieving the Dream 


95.9 


59.0 


2.1 


Other grant programs (e.g., Title III) 


95.2 


45.4 


1.9 



* “Don’t Know” responses treated as missing. 
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Accessibility of Data and 
Perceived Barriers to Data Use 

Sources of data on students. Faculty and 
administrators at the Achieving the Dream colleges 
indicated that they use a variety of sources or methods 
to get information on groups of students (see Table 
8). About a third of faculty members do searches 
themselves using their college’s student information 



system or their college’s website or fact book. Similar 
proportions make requests for information to the 
college’s IR or IT offices or use reports distributed by the 
college. Administrators are even more likely than faculty 
to search for information themselves, although the largest 
percentages make requests to IR or use reports from the 
college. Fewer than one in ten faculty members and one 
in four administrators indicated that they generally do not 
need information on groups of students. 



Table 8 

When 1 need information about groups of students (i.e., more than one student at a time), 
1 use the following souroes. (Respondents were asked to oirole all that apply.) 




Percentage of Respondent Group Indicating That 
They Use the Given Source 


Source 


Faculty 


Administrators 


Searches using the college’s student information system 


32.9 


41.1 


Data from the college’s website or fact book 


30.9 


43.2 


Reports distributed by the college’s institutional research (IR) office or 
other departments 


32.7 


50.0 


Requests to the IR or information technology (IT) staff 


32.1 


53.6 


My department’s database 


24.2 


29.5 


state databases or research reports 


22.3 


25.4 


1 generally do not need information about groups of students 


9.7 


23.1 



A majority of faculty and administrators seem satisfied 
that they are able to access information they need in a 
timely manner and that the information they receive is 
accurate (see Table 9, page 16). About half of faculty 
respondents (51.8%) indicated that the research reports 
and other information the college provides to faculty are 



generally helpful to their work as teachers. Only about a 
quarter (25.6%, not shown in the table) disagreed with 
the statement that such information is helpful to them in 
their teaching roles (the other quarter was neutral, neither 
agreeing nor disagreeing). 
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Table 9 

To what extent do you agree or disagree with the following statements? 




Percentage of Respondent Group Indicating 
They “Agree” to “Strongly Agree” 




Faculty 


Administrators 


The data in the college’s student information system are generaiiy 
accurate and error free. 


63.9 


70.7 


The data 1 need are generaiiy available in a user-friendly format. 


57.9 


62.1 


The college’s institutional research staff is responsive to requests 
for information. 


70.1 


77.5 


The college’s institutional research staff is adequately staffed for the college’s 
information and research needs. 


49.2 


52.9 


The reports and other information the college provides to administrators and 
faculty are typically clear and easy to follow. 


62.8 


72.3 


1 am able to obtain the information 1 need in a timely fashion. 


62.4 


69.1 


The research reports and other information the college provides to faculty 
are generally helpful to our work as teachers. 


51.8 


— 



* “Don't Know” and “Not Applicable” responses treated as missing. 



Reasons for not using data. As shown in Table 10, 
about a third of faculty indicated that one reason they 
do not use data and research is that they are too busy 
with their teaching responsibilities. Other than that, 
most faculty seem to have thought that using data and 
research on students is part of their responsibility as a 
faculty member and that they have the skills needed to 
analyze data. 



Over 30% of administrators Indicated that it is not their 
responsibility to use data and research on students. 
This may be because our sample of administrators 
Included some In athletics and other areas where 
Information on students is generally not used. The fact 
that a similar percentage of administrators Indicated 
that the data available are not relevant to their roles 
supports this conclusion. 



Table 1 0 

1 generally do not use data and research on students for the following reasons. 




Percentage of Respondent Group Indicating 
They “Agree” to “Strongly Agree” 


Reason 


Faculty 


Administrators 


1 am too busy with my teaching responsibilities. 


32.0 


8.7 


It is not part of my responsibilities as a faculty member/administrator. 


13.4 


31.0 


1 do not have the research skills to understand and use data and research. 


16.5 


10.6 


1 do not trust the data that are available. 


16.2 


11.2 


The data that are available are not relevant to my role as a faculty 
member/ administrator. 


21.4 


31.9 
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Relevant training. More than a third of faculty (35.2%) 
said that they have been involved in training or 
professional development in the past year on institutional 
research or data analysis, and over half (57.9%) said they 



have participated in training or development on program 
evaluation or assessment (see Table 11). This and the 
corresponding statistics for administrators were higher 
than we expected. 



Table 1 1 

Have you been involved in any training or other professional development in the past year related 
to the following topios? 




Percentage of Respondent Group Indicating “Yes” 


Topic 


Faculty 


Administrators 


Institutional research and/or data analysis 


35.2 


37.2 


Program evaluation and/or assessment 


57.9 


40.6 



4. Correlates of Data Use by 
Faculty and Administrators 

As is evident from the last section, the survey results 
reveal considerable variation in the extent to which 
individual faculty members and administrators at 
Achieving the Dream colleges use data in their jobs. This 
section examines factors associated with the tendency 
of faculty and administrators to use (or not use) student 
data in their work. The factors examined include, among 
others, attitudes about data use, perceptions of data 
accessibility and usefulness, department and college 
policies, and participation in Achieving the Dream. Similar 
analyses were conducted on factors associated with the 
extent to which faculty and administrators participate 
in organized discussions on ways to improve student 
achievement. 

Indicators of Data Use 

Both the faculty and administrator surveys contained 
multiple questions about the use of data and 
participation in organized discussions related to 
improving student success. To simplify the analysis of 
factors associated with the behaviors of interest, we 
created four composite indicators of data use by faculty 
and administrators. 



1 . Use of data on student outcomes. 

2. Use of data disaggregated by student race, ethnicity, 
or income. 

3. Participation in organized discussions on improving 
student success. 

4. Use (by faculty) of data and research in teaching- 
related decisions. 

The full report of this study lists the specific survey items 
used to create each indicator and outlines the method 
we employed to do so. It also includes tables showing 
the correlations among the four indicators for faculty and 
among the three indicators for administrators. Generally, 
the correlations among the indicators are fairly strong. 
Faculty and administrators who use student outcomes 
data frequently also use disaggregated student data and 
participate in organized discussions about improving 
student achievement more frequently. The correlation is 
somewhat weaker, though still positive, between use by 
faculty of data in teaching decisions and the indicators 
of use of outcomes data and participation in discussions 
about improving achievement. 

To get a sense of why some faculty and administrators 
might use data on student outcomes and engage in 
discussions about improving student achievement 
more than others, we calculated the correlations 
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between the four composite indicators of data 
use described above and various factors that we 
hypothesized are associated with use of data by the 
faculty and administrators in our sample. These factors 
can be grouped in the foiiowing categories. 

• Involvement in Achieving the Dream 

• Perceived usefulness of student data 

• Perceived accessibility and quality of student data 

• Reasons for not using data 

• Attitudes about the potentiai of students to 
succeed and the coilege’s responsibility in helping 
students succeed 

• Participation in training on data analysis, 
assessment, or program evaluation 

• Academic department practices on use of data and 
program evaluation and improvement 

• Coilege/administrative department practices and 
ieadership for data-driven decision making and 
student success 

• Influence of external factors in promoting data use by 
college administrators (administrator survey only) 

• Academic program area (for facuity) or administrative 
functionai area (for administrators) 

• Length of time at the coliege, age, and 
(for facuity) rank 

• Respondent’s demographics. 

Findings on Correlates of Data Use 

In the text that follows, we examine the correlations 
between the factors listed above and the four indicators 
of data use. Note that the full report of this study 
includes a table of findings on the correlations between 
various factors we hypothesized to be related to data use 
and the four indicators of data use. Detailed statistics 
on correlations (for continuous factor measures) and 
differences of mean responses (for categorical measures) 
are presented in an Excel document that is available from 
the authors. 

We want to stress that in this analysis we are merely able 
to examine the correlation between particular factors 
and the indicators of data use. This analysis cannot tell 
us if the relationship is causal. So, for example, although 
we find that participation in training on data analysis and 
program assessment is positively correlated with the 



indicators of data use, this does not necessarily mean 
that colleges can increase data use by increasing the 
amount of training provided. It could be that faculty and 
administrators who are heavier users of data are more 
likely to seek out training in data use. 

Involvement in Achieving the Dream. Faculty 
and administrators who were involved in Achieving 
the Dream, either in the core or data teams or in 
implementing strategies, used data on student 
outcomes more frequently and participated in organized 
discussions on improving student outcomes much 
more frequently than did their colleagues who were not 
involved in the initiative. This is not surprising given 
that colleges participating in Achieving the Dream 
are encouraged to broadly engage faculty, staff, and 
administrators in examining data on student progression 
and outcomes, and to work together to design, 
implement, and evaluate efforts to improve student 
achievement, particularly among groups of students who 
have faced barriers to success in the past. 

Faculty members participating in Achieving the Dream 
were much more likely than non-participating faculty to 
make more frequent use of data that are of particular 
interest to Achieving the Dream. Thus, for example, 
the differences between participating and non- 
participating faculty in the frequency with which they 
reviewed data on developmental education completion 
rates and academic achievement gaps are greater 
than the differences for other student outcomes data 
for measures that are not central to Achieving the 
Dream. A similar pattern is evident in the rates of 
student outcomes data use between participating 
and non-participating administrators. Participation in 
Achieving the Dream is also associated with a greater 
tendency of faculty and administrators to use data 
disaggregated by race or ethnicity and by income or 
receipt of financial aid. 

We hypothesized that faculty members and 
administrators who believed that Achieving the Dream 
would have a lasting impact on college practice would be 
more likely to use data than would faculty who believed 
that Achieving the Dream is just another “flavor of the 
month” whose effects will fade quickly when the grant 
ends. However, we found only weak correlations (< 0.23) 
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between measures of faculty and administrator opinions 
about the sustainabiiity of Achieving the Dream and the 
indicators of data use. 

As mentioned above, these results simply show 
correlation, not causality. Thus, although we found that 
participation in Achieving the Dream is correiated with 
greater frequency of data use, we cannot say definitively 
that involving more faculty and administrators in Achieving 
the Dream will result in higher rates of data use. 

Perceived usefulness of student data. Not surprisingly, 
faculty and administrators who had positive views about 
the usefulness of various sorts of student outcomes data 
(such as learning measures, retention and graduation 
rates, and achievement gaps) used student outcomes 
data more frequently than did those who question the 
usefulness of such data. Faculty who viewed data 
positively were specifically more likely to use data and 
research to inform decisions related to their teaching. For 
faculty, the correlation between the perceived usefulness 
of student outcomes data and the measure of the use of 
data in teaching-related decisions is especially strong. 

Perceptions of the usefulness of student outcomes data 
are also correlated with the frequency of participation 
by faculty and administrators in organized discussions 
with peers about improving student achievement and the 
frequency of using data disaggregated by race, ethnicity, 
or income, but the correlations are weaker than those 
associated with indicators of the frequency of data use. 

Faculty who believed that institutional research Is useful 
to their work as teachers were somewhat more likely to 
use data and participate in discussions about improving 
student outcomes than were faculty who thought 
otherwise, but the correlation is not strong (ranging from 
0.21 to 0.34 across the indicators of data use). 

Perceived accessibility and quality of data on 
students. Surprisingly, there is little correlation (< 0.2) 
between the perceptions of faculty and administrators 
about whether or not data on students were accurate 
and accessible and the frequency with which they used 
data on student outcomes or participated in organized 
discussions on improving academic achievement. This 
suggests that faculty and administrators who use data 



are motivated to do so regardless of whether the data are 
readily accessible or not. It is also possible that faculty 
and administrators get their data from the institutional 
research office and therefore do not have to deal with 
the issue of data accessibility. However, this explanation 
alone does not account for the weak correlation between 
perceptions of accuracy and frequency of data use. 

Reasons for not using data. The survey asked faculty 
and administrators about various possible reasons 
they might not use data on students, such as being 
too busy, not feeling responsible for analyzing student 
data, or not having the skills to do so. For both faculty 
and administrators, those who felt more strongly about 
reasons they might not use data were, as expected, less 
likely to use data as measured by all four indicators. 

The reasons most associated with a lower tendency 
to use data were that using student data was not a job 
responsibility or that the data were irrelevant to their 
jobs. This was even more the case for administrators 
than for faculty. Perhaps this is the result of the fact that 
we included in our survey all senior level administrators, 
including those in areas such as athletics and finance, 
who may have been less concerned with student data 
as part of their jobs than administrators in academic or 
student affairs. 

Faculty and administrators who indicated that they do 
not have the skills to analyze data or that they do not 
trust their college’s data were also less frequent users 
of data, although the correlations are weak. Similarly, 
faculty who indicated that participating in discussions 
about improving student success was not a good use 
of their time were less frequent users of data on all four 
indicators, although here again the correlation is weak. 

Attitudes about the potential of students to 
succeed and the college’s responsibility in helping 
students succeed. We hypothesized that faculty and 
administrators would be more likely to use data on 
students if they believed that their college could actually 
do much to improve student outcomes and should be 
doing so. However, we found only a weak correlation 
between the four indicators of data use and faculty and 
administrator beliefs about the potential of all students 
to succeed academically. We found no correlation 
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between the indicators of data use and the extent to 
which respondents believed that it is appropriate for 
their coilege to provide extra help to certain groups of 
students or felt that they and their colleagues could do 
more to help students succeed. 

Length of time at the college, age, and (for faculty) 
rank. We hypothesized that junior facuity wouid be 
more iikely than senior facuity to use data on student 
outcomes because junior faculty might be more 
comfortable with technology and because they are still 
learning what is most effective for them in the ciassroom. 
However, the oniy statisticaliy significant difference 
we found across the four indicators of data use is that 
assistant professors used data on student outcomes less 
frequently than did fuli professors. 

Also contrary to expectation, there does not seem to be 
a correiation between the length of time facuity members 
or administrators have been at their coiiege or their age 
and their use of student data. 

Surprisingly, administrators who were over 55 were 
much more frequent users of student outcomes data 
than those under 35. Administrators over 55 were also 
more frequent users of student data disaggregated by 
race, ethnicity, or income. Administrators over 35 were 
more iikely than those under 35 to participate frequently 
in organized discussions about improving student 
achievement, interestingiy, administrators 65 or oider 
were most likeiy to participate in discussions about 
improving student achievement. 

Gender and race/ethnicity. Femaie faculty members 
indicated that they used data and research in teaching- 
reiated decisions more frequently than did maie faculty 
members and participated in organized discussions 
about students more frequentiy than did their maie 
counterparts. However, there is no statisticaliy significant 
difference in the frequency with which maie and female 
faculty used data on student outcomes. 

White facuity were less likely than non-White faculty to 
frequently use data on student outcomes, to participate 
in organized discussions on student achievement, to 
use data and research in teaching-reiated decisions, 
and to use data disaggregated by race, ethnicity, 



and income. Facuity who ciassified themselves as 
Black were more frequent participants in organized 
discussions on improving student achievement. Biack 
and Native American facuity used data broken out by 
race, ethnicity, and income more frequently than did 
faculty of other races and ethnicities, interestingiy, 
Asian/Pacific islander faculty used data in teaching- 
reiated decisions more frequentiy than did facuity of 
other racial and ethnic groups. It is also notable that 
there does not seem to be any correlation between 
Hispanic facuity and use of data. 

in contrast to the patterns observed for faculty, White 
administrators were more frequent users of student 
outcomes data than were non-White administrators. 
Hispanic administrators were iess iikely to participate in 
organized discussions on improving student achievement 
than were non-Hispanics. 

Participation in training on data anaiysis, assessment, 
or program evaiuation. Faculty and administrators 
who had received training or professional development 
on analyzing data or on program evaiuation were, not 
surprisingly, more likely than faculty members who had 
not received such training to use data across ali the 
indicators. Again, our anaiysis oniy shows correiation, not 
causaiity, so it is not necessarily the case that colleges 
will increase use of data if they train more faculty and 
administrators on data use and program assessment. 
That could happen, but it could also be the case that 
faculty and administrators who are already motivated to 
use data pursue training that they think will aid them in 
doing so. 

Academic department practices on use of data 
and program evaluation and improvement. Facuity 
in academic departments that use data and research 
in departmental decision making themselves, not 
surprisingly, used data and research more frequently 
in teaching-reiated decisions than did facuity in 
departments where data-based decision making is not 
so prevalent. Faculty in departments that use data for 
decisions more extensively also used data on student 
outcomes and participated in organized discussions 
on improving student success, aithough with these 
indicators, the correlation with department practices is 
not as strong. 
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Similarly, faculty in departments that meet at least once a 
year to discuss student success rates and that evaluate 
the effectiveness of programs were more likely to use 
data as measured by the four indicators employed here. 
The mean responses indicate that faculty who reported 
that their departments meet at least once a year to 
discuss the success rates of their students reviewed and/ 
or used student outcomes data at least once every two 
to three years, while those in departments that meet less 
frequently used student outcomes data less frequently. 
Meeting with colleagues as a department at least once 
a year, not surprisingly, is correlated with more frequent 
participation in organized discussions about academic 
achievement and a greater tendency among faculty to 
use data and research on students in their own decisions 
about teaching. 

Academic program area (for faculty). Faculty in 
general education were on average less likely than 
faculty in other program areas to use data on student 
outcomes and to use data and research in decisions 
related to their teaching on a frequent basis. In contrast, 
faculty who teach in developmental or for-credit 
occupational programs were more likely than those in 
other fields to do so. Developmental faculty members 
were also significantly more likely to participate in 
organized discussions on student achievement and to 



use data disaggregated by race, ethnicity, or income. 
Adult basic education faculty used data no more 
frequently than faculty in other areas. Interestingly, 
even though they were more likely than faculty in other 
areas to use data in teaching-related decisions, faculty 
in for-credit occupational programs were less likely 
to participate in organized discussions about student 
achievement or to use data broken down by race, 
ethnicity, or income. Faculty in non-credit occupational 
programs used data related to student outcomes more 
frequently than did faculty in other fields, but there are 
no statistically significant differences in the rates of data 
use between these faculty and others on the other three 
indicators of data use. 

Not surprisingly, we found that faculty members who 
also have administrative roles were more likely than 
faculty not involved in administration to use data across 
the four measures. 

Administrative functional area (for administrators). 

Table 12 ranks administrative functional areas by the 
three indicators of data use applicable to administrators. 

It is not surprising that institutional research 
administrators were the most frequent data users on all 
three indicators. Academic affairs administrators were 
more frequent users of data than were those in student 



Table 1 2 

How much do you use data and research on students in your own decisions about the following? 


Rank 


Frequency of use of student 
outcomes data 


Frequency of participation 
in organized discussions on 
improving student success 


Frequency of use of data 
disaggregated by race/ethnicity 
or income 


1 


Institutional research 


Institutional research 


Institutional research 


2 


Academic affairs 


President’s office 


External relations 


3 


President’s office 


Academic affairs 


President’s office 


4 


External relations 


Student affairs/services 


Academic affairs 


5 


Career and technical ed. 


Instruction 


student affairs/services 


6 


student affairs/services 


Career and technical ed. 


Admissions 


7 


Instruction 


External relations 


Career and technical ed. 


8 


Admissions 


Admissions 


Instruction 


9 


Continuing education 


Registration 


Finance 


10 


Information technology 


Athletics 


Continuing education 
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affairs/services. (It is not clear how “academic affairs” 
differs from “instruction.”) Interestingly, administrators 
in external affairs were more frequent users of data 
on student outcomes than were student affairs 
administrators. Externai affairs administrators were also 
more likely to disaggregate data by race, ethnicity, or 
income than were administrators in both student and 
academic affairs. Administrators who also had a faculty 
role were more likely to use data on all three measures. 

College/administrative department practices and 
leadership. There is only a weak correlation between 
the various indicators of data use by individual faculty 
and the extent to which respondents indicated that 
their college overall uses data on student outcomes 
to evaluate programs. Even weaker is the correlation 
between faculty data use and faculty members’ 
perceptions about the level of commitment by the 
college’s leadership to making decisions based on 
data and the clarity of the leadership’s vision on how to 
increase student academic success. These findings and 
the earlier ones about departmental practices suggest 
that the practices of individual academic departments 
have a greater bearing on the use of data by faculty 
members than do those of the college overall. 

There is also surprisingly little correlation between the 
extent to which administrators said that their college 
uses data for program-related decisions and the 
frequency with which they themselves used data. 

Administrators at institutions that evaluate the 
effectiveness of educational programs and services or 
that use data on student outcomes to evaluate academic 
programs and departments used data and engaged 
in organized discussions about improving student 
achievement more frequently than did administrators at 
colleges that do not evaluate programs and services, but 
the differences are not statistically significant. Similarly, 
there does not seem to be much correlation between 
the frequency with which administrators used student 
outcomes data or engaged in organized discussions on 
improving student outcomes and whether or not: 

• The college has a strategic plan; 

• Departments and divisions are required to set 
measurable goals as part of the planning process; 



• The college requires that budget requests be 
supported by evidence that students will benefit. 

Curiously, the three practices above are associated 
with a lower frequency of use among administrators of 
student data disaggregated by race, ethnicity, or income. 
It is not clear why colleges with established systems for 
strategic planning and budgeting would be less likely to 
use data disaggregated by race or income than would 
colleges without such systems. 

Influence of external factors in promoting data use by 
college administrators (administrator survey only). We 
asked administrators about the extent to which various 
internal and external influences have promoted the use of 
data by the college. These included college leadership, 
boards of trustees, state mandates, accreditation 
reviews. Achieving the Dream, and other grant programs. 
Surprisingly, there is only a very weak correlation 
(< 0.1) between administrators’ ratings of the influence 
of these factors on the use of data by administrators at 
the college and their own use of data. 

Discussion of Correlates 

These findings are encouraging for initiatives designed 
to bring about changes in behavior among members of 
an institution. Again, while the findings do not establish 
a causal relationship, we did see a few persistent 
correlations that are compelling. For example, faculty 
members and administrators participating in Achieving 
the Dream were much more likely than non-participating 
faculty to make more frequent use of data that are of 
particular interest to Achieving the Dream. The survey 
also asked faculty and administrators about various 
possible reasons they might not use data on students. 

We found that the reasons most associated with a lower 
tendency to use data were that using student data was 
not a job responsibility or that the data were irrelevant 
to their jobs. Colleges engaged in such an initiative may 
thus want to discuss the role that reviewing student data 
plays in faculty and administrators’ responsibilities and 
may want to clarify the ways in which the available data 
are relevant to various jobs at the college. 

Also notable were the differences in frequency of data 
use on student outcomes, participation in organized 
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discussions on student achievement, use of data and 
research in teaching-related discussions, and use of 
disaggregated data by faculty and administrators of 
different races/ethnicities. 

The differences in data use by department, in particular, 
are interesting. We noted that faculty who teach in 
developmental or for-credit occupational programs were 
more likely to use data and research to inform teaching- 
related practices than those in other fields. And we noted 
that faculty in non-credit occupational programs used 
data related to student outcomes more frequently than did 
faculty in other fields. The finding for developmental faculty 
is consistent with the focus of Achieving the Dream. It may 
be that the findings for the for-credit occupation programs 
result from longstanding licensure and certification 
requirements— data on these credentials may impact what 
these faculty do in the classroom. Our findings related to 
non-credit occupational programs may be the result of 
these faculty acting in a more entrepreneurial and market- 
oriented fashion as they create course offerings. In doing 
so, they may rely more heavily on data. 

The fact that the correlation between academic 
department practices and faculty practices was stronger 
than that between the college’s overall use of data and 
the faculty’s use of data was unexpected and requires 
more exploration. In addition, we hypothesized that we 
would see a meaningful positive relationship between 
administrators’ use of data and administrators who 
responded that their college engaged in program 
evaluation and strategic planning, required the 
departments and divisions to set measurable goals as 
part of the planning process, and tied budget requests to 
evidence that students will benefit. Instead, we found that 
these practices are associated with a lower frequency of 
data use among administrators. Additional analyses are 
needed to better understand these relationships. 

Finally, as we discussed in our methodology section, we 
selected the states based on a conception that there 
was significant pressure on the colleges from accrediting 
agencies and state policy to use data to improve student 
academic success. We hypothesized that this pressure 
would drive administrators to use data. However, we 
found only a very weak correlation (< 0.1) between 
administrators’ ratings of the influence of these factors 



on the use of data by administrators at the college and 
their own use of data. 

5. Patterns of Data Use by College 

In this section, we report briefly on our examination of 
the amount of data use among the Achieving the Dream 
colleges in our sample. To do this, we ranked the colleges 
according to their average scores on the four indicators 
of data use by faculty and three indicators of data use by 
administrators. Recall that the indicators of data use that 
we developed are: 1) use of data on student outcomes; 

2) use of data disaggregated by student race, ethnicity, 
or income; 3) participation in organized discussions on 
improving student success; and 4) use (by faculty) of 
data and research in teaching-related decisions. We did 
a similar ranking by the round in which colleges joined 
Achieving the Dream (Round 1 or Round 3), and the state 
in which they are located. Our aim was to determine 
if there is consistency in the ranking of colleges by 
the various indicators of data use. If we did find some 
degree of consistency in the ranking of colleges by these 
indicators, this would give us confidence to use them as 
measures of the amount of data use at the institution level. 
It would allow us to examine the relationship between 
college-level indicators of the amount of data use and 
measures of institutional performance. We could also 
examine the practices of colleges with higher levels of 
data use by faculty and administrators with those where 
the amount of data use is lower. To the extent that there is 
not consistency in the ranking of colleges by these various 
measures, it would be difficult to use these indicators 
appropriately for comparison. 

The full report of this study includes tables that show 
the mean values for each college of the indicators of 
data use as well as rankings of the colleges by the 
mean value of each indicator. We found considerable 
variation in the mean responses across the colleges. 

There are many cases in which a college’s ranking varies 
considerably across the different indicators. For example. 
College 34 ranks third on faculty indicator 1 , thirty-fifth 
on faculty indicator 2, thirty-first on indicator 3, and 
eighteenth on indicator 4. A similar lack of consistency in 
rankings is evident across the indicators of data use by 
administrators. Moreover, the rankings of colleges by the 
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comparable indicators for faculty and administrators do 
not in general follow a clear pattern. For example, College 
26 ranks second in the indicator of the frequency with 
which faculty use data on student outcomes and thirty- 
third on the comparable measures for administrators. This 
is consistent with the finding from the previous section 
that there does not seem to be a strong correlation 
between the extent to which faculty use data and research 
and their perceptions of the extent to which the college 
generally uses data for decision making. 

Another finding from the previous section is that faculty 
and administrators who participated in Achieving 
the Dream are more likely than those who had not 
participated to use data according to the four indicators. 
Given this, we hypothesized that colleges where a greater 
percentage of faculty or administrators were involved in 
Achieving the Dream would on average exhibit higher 
rates of data use. Yet a lack of rank order consistency 
is evident between the measures of the extent to 
which both faculty and administrators were involved 
in Achieving the Dream and indicators of data use by 
faculty and administrators. We also hypothesized that 
it would be easier to engage faculty and administrators 
in using data for decision making at smaller colleges 
compared to larger ones because of the challenges of 
communication and coordination in large organizations. 
However, there does not seem to be a clear correlation 
between institutional size (measured by FTE enrollment) 
and indicators of data use by faculty and administrators. 

A table included in the full report of this study shows 
the mean institutional-level values for the faculty and 
administrator indicators by state and by the round in 
which colleges joined Achieving the Dream. Another 
table ranks the states and rounds by these mean values. 

The results show that colleges in North Carolina rank 
higher than those from other states in the average values 
for the first three indicators of faculty data use, but third 
on the indicator of the extent to which faculty use data 
on students in teaching-related decisions. The Achieving 
the Dream colleges in Texas rank highest (on average) on 
all three indicators of administrator data use. However, 
there is a fair amount of inconsistency in the ranking of 
the other states across the various indicators of data use 
for both faculty and administrators. 



The colleges that joined Achieving the Dream in the 
first round on average exhibit higher rates than those 
in the third round of data use on all of the faculty and 
administrator indicators, with the exception of the 
measure of the extent to which faculty participate in 
organized discussions on improving student success. 
This is consistent with the hypothesis that colleges that 
have been involved in Achieving the Dream longer will be 
more advanced in their use of data for improving student 
success. However, these findings are merely suggestive; 
they cannot be seen as definitive evidence of a causal 
relationship between Achieving the Dream and more 
extensive use of data for improvement. 

6. Findings from Telephone 
Interviews 

This section presents findings from telephone interviews 
of faculty and administrators at four of the Achieving 
the Dream community colleges that participated in the 
survey. In total, we interviewed 17 faculty members 
and 25 administrators/staff. The interviewees included 
a mix of individuals who had and had not been directly 
involved with the Achieving the Dream initiative. The 
faculty we interviewed taught either developmental or 
college-level math or English. The senior administrators 
we interviewed included, at each college, the senior 
academic officer, the senior student services officer, the 
director of institutional research, the Achieving the Dream 
coordinator, and at least one member of the student 
services staff. 

We analyzed the interview transcripts to look for 
themes that could further explain the survey findings. 

In particular, in this section we address data use at the 
colleges and the connection (or absence of a connection) 
between strategic planning and budgeting and the use of 
data by faculty and administrators. 

Use of Data by Faculty and Administrators 

In general, the telephone interviews we conducted 
reinforce the survey findings about the use of data. 
Faculty and administrators reported that they reviewed 
and used student outcomes data on a regular basis. 

In the survey we asked specifically about grades. 
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placement scores, retention rates, transfer rates, 
and graduation rates. We also asked about research 
carried out by the college and about college-based or 
outside research on effective practices. In more open- 
ended discussions about data use in the interviews, 
we heard many faculty refer, not surprisingly, to 
grades and informal in-class assessments of student 
comprehension of course material as measures of 
student performance. To a more limited extent, faculty 
also spoke about using placement scores and scores 
on other tests to gauge the preparedness of their 
students and to understand the range of skills they 
possessed. The administrators we interviewed generally 
discussed the use of data on enrollments, retention 
rates, and graduation rates. Administrators also often 
mentioned data types that we did not ask about in the 
survey, including information that supports course and 
faculty scheduling, tracking data on tutoring center 
usage, and health services statistics. 

Faculty interviewees at three of the four colleges 
reported that they regularly met with colleagues to 
discuss student achievement in their courses and 
to compare perceptions and trends about student 
progression. Two interviewees reported on how such 
discussions eventually facilitated a change in a specific 
practice (in one case, the result was a faculty-wide push 
to improve critical thinking skills among students across 
the curriculum; in the other case, the result was the 
prohibition of the use of calculators in developmental 
math courses). While the evidence used during the 
discussions that led to these changes reportedly 
emphasized first-hand observations of students more 
than numerical data, in both cases placement test 
scores were also used to understand the level of 
students who were entering classes. 

We heard a range of perspectives on the role of faculty 
in using student outcomes data. According to the 
survey results, only 13% of faculty did not use data 
and research on students because they believed it was 
not part of their responsibilities. While many faculty 
interviewees indeed argued that faculty should be 
actively involved in data collection and analysis efforts, 
several remarks made by others suggest that among 
other faculty there is a strong disinclination to collect or 
use such data. In one interview, for example, a professor 



who did not think that the use of data and research was 
part of a faculty member’s responsibilities questioned 
the value of data, arguing that many faculty members, 
who as teachers have nearly daily exposure to their 
students in classroom settings, are in a good position to 
understand those students’ needs without recourse to 
additional data. 

The interview findings suggest that administrators 
and faculty who sit on committees may be more likely 
than others to use large-scale surveys and tests. For 
example, administrator interviewees at two colleges 
cited the use of internally developed surveys to gather 
feedback from faculty and students on their opinion 
of areas needing improvement. Interviewees from 
three colleges noted the use of external instruments, 
including CCSSE, Noel-Levitz, or the University of 
Texas’ Learning and Study Strategy Inventory (known 
as LASSI, this is a tool developed at the University of 
Texas at Austin that assesses students’ “awareness 
about and use of learning and study strategies related 
to skill, will, and self-regulation components of 
strategic learning”), to identify areas of improvement. 
Using LASSI data as well as registration and course 
completion data and grades, one college developed a 
model of students at risk showing that students who 
fail or drop out of the same course twice have very low 
success rates. They then developed a plan requiring 
that such students see an academic advisor for the 
provision of additional supports and services prior to 
registration for additional courses. 

It is clear that all four of the colleges involved in the 
telephone interviews make use of a variety of data 
sources, but the extent to which they are using data they 
themselves collect to improve practices varies. We did 
hear from interviewees at one college about their having 
implemented evaluations of strategies they pursued and 
about how they have now acquired preliminary evidence 
suggesting that their strategies result in improved 
learning outcomes. Interviewees at another college 
discussed the goal of using data to inform classroom 
instruction. They have developed software that allows 
faculty (who choose to do so) to enter course objectives 
and whether they meet those objectives. The ultimate 
impact of having that information and whether it results in 
changes in classroom practices remains to be seen. We 
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heard little from our interviews with representatives from 
any of the colleges suggesting that analysis of internaiiy- 
collected data is college-wide. 

Three of the four colleges are using “dashboards.” 

These dashboards provide current statistics to senior 
administrators on academic indicators such as enrollment 
and retention as weil as other kinds of information, 
including student visits to the college’s health services 
unit for specific reasons. The use of dashboards and fact 
books as sources of data contributes to a perception 
held by many of the interviewees that their college has 
established a culture of evidence. For example, with 
respect to their fact book, one administrator commented 
that “it is just a wealth of information for us. Now we 
are looking at that and saying, ‘Okay, I see that our 
graduation rates have started to slip a little bit.’ And 
we start talking about the data and why and what we 
can do to maybe change the course of that. So, yes, I 
would say that ‘data-driven’ is permeated throughout 
the college.” These dashboards represent a significant 
change in information access. First, they permit access 
to a wider range of information to a wider range of people 
at the college. Second, they provide current, sometimes 
in real time and sometimes daily, updates of key college 
indicators. Third, they represent a form of data that 
is “pushed out” to people at the college, rather than 
having individuals retrieve the information themselves. 
Clearly, as the quotations above indicate, the dashboards 
provide information on which administrators can act. 

Yet it is still unclear whether these dashboards are part 
of a continuous improvement process that involves 
resolving issues and evaluating solutions or whether they 
just provide information on college’s performance that 
is not systematically used in decision making about the 
management of programs and services. 

In general, the absence of more extensive and more 
complex data analysis may In part be due to problems 
of time, resources, and expertise. Although the survey 
results Indicate that only 32% of faculty and only 9% of 
administrators are too busy to use data and research 
on students, a number of interviewees made comments 
about either not having the time to use the considerable 
amount of data available at the college or not having 
the expertise. For example, one interviewee expressed 
interest in comparing the performance of students who 



have and have not received tutoring in courses that they 
take after participating in that intervention. However, she 
has not had the time to pursue this interest. 

Interviewees also said that they rely on their institutional 
research office to help them interpret the data they do 
receive, although at least one senior administrator at each 
college reported extracting and analyzing data themselves. 
A reliance on the IR staff can be problematic because IR 
staff are generally very busy and may not have the time to 
help everyone analyze relevant data. Many of the faculty 
and administrators we spoke with said they did not feel 
comfortable analyzing data or even know what questions 
to ask. This is inconsistent with our survey findings in which 
only 17% of faculty and 1 1 % of administrators indicated 
not having the skills to analyze data. 

The Relationship Between Strategic 
Planning and Budgeting and Use of Data 

One surprising finding from the survey was the weak 
correlation between the extent to which administrators 
and faculty used data on student outcomes and whether 
or not they indicated that their college overall uses data to 
evaluate programs or to make program-related decisions. 
Similarly, the frequency with which administrators used 
data did not seem to be related to whether or not college 
departments were required to set measurable goals as 
part of the planning process or whether or not the college 
required that budget requests be supported by evidence 
that students would benefit. We expected that faculty 
and administrators in colleges where program review, 
strategic planning, and budgeting are informed by data on 
students would be more likely to use data themselves. Our 
telephone interviews suggest a possible explanation why 
we did not find this to be the case. 

The respondents to our telephone interviews indicated 
that involvement in strategic planning and budgeting 
is generally limited at their colleges. When faculty 
and lower-level staff are involved, it seems as though 
they are still far removed from the final budgeting 
decisions and setting of priorities. For example, while 
some colleges use large strategic planning teams 
that do involve personnel from across the college, it 
is not clear what influence these teams have on final 
budget allocations. Another model can be likened to a 
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pyramid in which plans get rolled up to the leadership 
group, which ultimately makes budget decisions. One 
interviewee described the process as follows: “Every 
two years we develop a set of Improvement objectives 
for every unit, and it sort of pyramids up so that all the 
units come together with the organizational entity above 
them and they come up with another prioritized list. And 
so, ultimately you end up with prioritized lists that are 
used to get funding. So if you want to get funded in the 
budget process, you have to show that it’s in your plan, 
which means you have to show that there’s some data to 
support the request, ... that’s how it gets linked. So you 
can’t get funded without a plan.” This college has clearly 
sought to use data to Inform planning and budgeting, but 
it is not clear how the data Inform the final decisions. It 
may be true that departments and divisions cannot get 
funded without a plan (and data), but it is not clear what 
role data play In how funding decisions are made vis-a- 
vis a number of competing plans. 

It is still a challenge for colleges to fully Integrate their 
planning and budgeting processes and use data to 
support those processes. Although there may be 
much talk about the use of data to inform budgeting, 
our interviews suggest that the reality is usually quite 
different. One administrator interviewee commented that 
although his college’s administration is trying to tie the 
budget and strategic planning processes together, “it’s 
a chore to try to get planning ahead of budgeting.” He 
added that budgeting has “not aligned itself very well 
with strategic planning.” 

These examples may help explain why the survey 
findings indicate that there is little correlation between 
data use by individual faculty and administrators 
and strategic planning and budgeting by the college. 
Interviewees at all the colleges stated that they have 
Integrated planning and budgeting so that the two 
processes do not occur separately and that the college’s 
priorities direct the budget, but they also qualified those 
statements, explaining that they were still improving the 
process and were working on further aligning planning 
and budgeting. 

One area where there was clear consensus on the use 
of data was in making decisions about hiring practices, 
mostly as It relates to the ability to fund a new position or 



to override a restriction in terms of a hiring freeze. Data, 
such as enrollment trends, are needed to justify a new 
position or why a vacant position must be filled. 

Findings from the interviews suggest that data use for 
program review is mixed. It appears that there may be 
a lot of unfocused data use in program reviews. For 
example, one faculty member commented, “When the 
committees or departments present the program review, 
oh my goodness, [there] is so much data in there.... 
fTjhere is too much information. Information overload.... 
It’s just all these charts....” Another professor said that 
he didn’t see overall program assessment as his job; 
he felt that the student outcomes for his courses were 
where his responsibilities ended. 

7. Conclusion 

This report summarizes findings from what is to our 
knowledge the most extensive survey to date of the 
use of data and research for decision-making by 
faculty and administrators at community colleges. It is 
based on surveys of faculty and administrators at 41 
colleges involved with Achieving the Dream as well as 
telephone interviews with faculty and administrators 
at four of those same colleges. Achieving the Dream 
is a national initiative designed to improve outcomes 
for community college students, particularly students 
of color and low-income students. One of the key 
principles of Achieving the Dream is that for colleges 
to improve student outcomes on a substantial scale, 
they need to engage faculty, administrators, and staff 
on a wide scale in examining data on how students 
perform and in using that information to devise 
strategies for addressing gaps in attainment among 
student groups. Colleges should seek to build a 
“culture of evidence” In which decisions about how to 
organize, manage, and fund instruction and student 
support services are made based on evidence of what 
works to promote student success. 

The survey showed that relatively high proportions 
of the faculty and administrators in the Achieving 
the Dream colleges surveyed used data on student 
outcomes, and the telephone interviews support this 
finding. More than half of faculty members reviewed or 
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used data on placement test scores, retention rates, 
graduation rates, and measures of student learning 
other than grades at least once a year. About half 
of faculty and administrators used data on student 
achievement gaps on an annual basis. About two thirds 
of faculty and administrators used outside research on 
effective practices at least annually. 

The majority of faculty at the Achieving the Dream 
colleges surveyed used data and research at least to 
some extent in decisions related to teaching. And most 
faculty and administrators surveyed indicated that they 
participate at least once a year in organized discussions 
on improving student success. Two thirds of faculty 
agreed that asking faculty to regularly participate in 
discussions about data on student outcomes is a good 
use of their time. Only 13% of faculty members indicated 
that using data and research on students is not part of 
their responsibilities as a faculty member, though the 
telephone interviews suggest that resistance to using 
data may be strong among some faculty. 

The high rate of data use at the colleges surveyed 
is perhaps not surprising, given their involvement In 
an Initiative premised on the importance of broadly 
engaging faculty and staff in using data and research to 
improve the quality of programs and services. Indeed, 
we found that faculty and administrators who were 
involved In Achieving the Dream used data on student 
outcomes more frequently and participated In organized 
discussions on improving student outcomes much more 
often than did their colleagues who were not Involved 
In the initiative. 

At the same time, not all participants in Achieving the 
Dream were heavy data users, and more generally, there 
was considerable variation among individual faculty 
and administrators in the extent to which they used 
student data. For example, over a third of faculty never 
used measures of student learning other than grades or 
looked at information on students broken down by race 
or ethnicity, and nearly one third never reviewed data on 
student achievement gaps. And the telephone interviews 
suggest that, despite survey findings that seem to 
indicate otherwise, many faculty and administrators 
may feel that they lack the skills to analyze data in ways 
meaningful to their jobs. 



Female faculty members indicated that they used 
data and research in teaching-related decisions more 
frequently than did male faculty members, and they 
participated in organized discussions about students 
more frequently than did their male counterparts. 

White faculty were less likely than non-White faculty 
to use data on student outcomes generally, to use 
data in teaching-related decisions more specifically, 
and to participate in organized discussions on student 
achievement. Faculty who classified themselves as 
Black and Native American used data disaggregated 
by race, ethnicity, and income more frequently than 
did faculty of other races and ethnicities. Interestingly, 
in contrast to patterns observed for faculty. White 
administrators were more frequent users of student 
outcomes data than were non-White administrators. 
Contrary to expectation, we did not find much of a 
correlation between a faculty member’s rank or the 
length of time a faculty member had been working at a 
college and the extent to which he or she used data. 

We found that the practices of faculty members’ 
departments were more strongly associated with their 
tendency to use or not use data than were practices 
and values of the college overall. Not surprisingly, 
faculty in departments that used data in departmental 
decision-making and that met frequently as a department 
to discuss student success rates and evaluate the 
effectiveness of programs were more likely to use data 
as measured by the four indicators of data use that we 
employed. In contrast, there is only a weak correlation 
between the various indicators of data use and the extent 
to which respondents indicated that their college overall 
uses data on outcomes to evaluate programs. 

There is also surprisingly little correlation between 
the extent to which administrators said that their 
college uses data for program-related decisions and 
the frequency with which they themselves used data. 
Similarly, there does not seem to be much correlation 
between the frequency with which administrators 
used student outcomes data or engaged in organized 
discussions on improving student outcomes and 
whether or not their college requires departments and 
divisions to set measurable goals and objectives as 
part of the planning process. The telephone interviews 
suggest that colleges continue to be challenged In their 



28 



efforts to fully integrate their planning and budgeting 
processes and encourage widespread use of data to 
support those processes. 

The apparent disconnect between the extent of data 
use by faculty and administrators and the views and 
management practices of the college’s leadership calls 
into question a central premise of Achieving the Dream — 
that commitment by a college’s leadership and the way 
that a college approaches program evaluation, strategic 
planning, and budgeting are key to encouraging the 
use of data for improvement by college personnel. Our 
findings suggest that leadership commitment and a data- 
oriented approach to institutional management may not 
be sufficient to encourage faculty and administrators to 
become more data-oriented in practice. Additional efforts 
at the department level are probably needed to change 
the behavior of faculty in particular. Indeed, we found 
that faculty in developmental education departments 
and for-credit occupational programs were more 
frequent users of data than were faculty in other types of 
departments, particularly those in general education. The 
greater intensity of data use in developmental education 
departments Is perhaps not surprising given that 
improving developmental Instruction has been a major 
focus of Achieving the Dream. The baseline evaluation of 
the first-round Achieving the Dream colleges found that 
the vast majority of participating colleges. If not all 
of them, were implementing some sort of strategy aimed 
at improving developmental outcomes (Brock et al., 

2007). It may well be that a similar intensive focus on 
improving outcomes is needed to change practices and 
to influence the culture in other types of departments. 

In examining the relative extent of data use among the 
colleges in our sample, we did not find much consistency 
in the rankings of these colleges by institution-level 
averages of the various indicators of data use by faculty 
and administrators. This suggests that colleges in the 
sample differed in the types of data they emphasized. 
Contrary to expectation, faculty and administrators at 
larger colleges were not, on average, heavier users of 
data than faculty and administrators at smaller colleges. 

Although colleges with higher levels of participation in 
Achieving the Dream by faculty and administrators did 
not exhibit higher average rates of data use, we did find 



that colleges that joined Achieving the Dream earlier 
(in Round 1 rather than Round 3) had higher rates of 
data use on all but one of the faculty and administrator 
measures. This is consistent with (although it does 
not prove) the hypothesis that colleges that have 
been involved in Achieving the Dream longer are more 
advanced in their use of data for improving student 
success. It also suggests that engaging faculty and staff 
in using data and building a culture of evidence is a 
complicated process that requires concerted effort over a 
long period of time. 

The findings from this study suggest three broad 
conclusions and one point for further analysis. First, the 
survey findings suggest that Achieving the Dream may 
have had an impact on data use at the colleges. The 
data use may not be as widespread as had been hoped 
or as integrated with planning and budgeting functions 
as might be expected, but the greater use of data on 
student outcomes by faculty and administrators who 
are involved in the initiative suggests that an externally 
originated initiative can bring about changes in practice. 
This is promising for Achieving the Dream and other 
initiatives seeking organizational change to improve 
institutional performance and student outcomes. As 
mentioned, CCRC and MDRC plan to survey a panel 
of faculty and administrators who participated in the 
survey for this study again in 201 1 . This will provide 
more definitive evidence about whether or not Achieving 
the Dream is helping to foster increased use of data in 
decision making. 

Second, the findings suggest that producing substantive 
changes in culture and practice is a long process. The 
Achieving the Dream initiative began as a five-year effort. 
Our finding that the Round 1 colleges, which joined in 
2005, tended to have higher rates of data use than the 
Round 3 colleges, which joined in 2007, indicates that 
several years of effort may be needed before changes 
in behavior are institutionalized. The follow-up survey 
planned for 201 1 will provide a clearer indication of how 
long this change process takes. 

Third, the apparent disconnect between the extent of 
data use by faculty and administrators and the views 
and management practices of the college’s leadership 
suggests that leadership commitment and a data- 
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oriented approach to institutional management may not 
be sufficient to encourage facuity and administrators to 
become more data-oriented in practice. Greater emphasis 
may be needed at the department level to encourage the 
use of data for improvement. This implies that the notion 
of broad engagement in the process of anaiyzing student 
data is particularly important, especiaiiy for department 
and division chairs at the college. 

Finally, further analysis is needed to better understand the 
relationship between data use and budgeting and planning 
efforts. Our finding that colleges with established systems 
for strategic planning and budgeting were less likely to 
use data disaggregated by race or income than colleges 
without such systems is counterintuitive; additional 
analysis may shed more light on this. 
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